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Introduction 


The  conditional  rate-distortion  function  has  proved  useful  in  source 
coding  problems  involving  the  possession  of  side  information  such  as  in 
simple  networks  and  sources  with  memory.  Many  of  the  basic  properties  of 
conditional  rates,  however,  are  lengthy  but  relatively  straightforward- 
extensions  of  the  usual  theory.  Hence,  these  results  have  not  appeared 
in  the  papers  concerned  with  the  applications  of  conditional  rates.  The 
purpose  of  this  report  is  to  present  these  basic  definitions,  coding 
theorems  and  bounds  so  as  to  provide  a  complete  background  reference  for 
the  journal  papers  on  composite  bounds  for  data  compression  performance 
attainable  with  sources  with  memory  [8]  and  on  source  coding  for  simple 
networks  [5] . 

Definitions  and  Preliminaries 

Joint  Rates 

The  joint  rate -distort ion  function  of  a  vector  source  U  = 

,  • . , U  )  and  a  vector-valued  distortion  measure  D(u,{i)  =  Jd,  (u,u),k=l, 

v  A 

...,m£  defined  on  A(U''  X  ft(U)  is  defined  as  follows: 

J  /v  r** 

Hrj<S)  =  inf  I(U;U) 

p^ulu)  e  yS) 

where 

■>,<»!“> « y« 


i 


if 


a)  Pt(u|u)  g  0 


b>  A  *  1 
^  /Tt\  ^ 


U€fl(U) 


C>  La 

uCA(U)  u€fi(U) 


s 

fiefl®)pt<£l2>S<2>V2»ii)  s 


k  — -  1 ,2, . . . ,m 


and 


„  V-  V  P+(u|u) 

I(U;U)  =  2  Z„  P* (**  ju)Q(u)  log-^v-- 

-  hJn/fix  t  -  uKO) 


ucA(U)  u€n(u) 


j(u)  s  y  P.  (u|u)Q(u) 

♦V  Lm—J  t  ^ 


Where  ft(U)  is  the  set  c £  letters  for  which  u>(u)  may  be  strictly 
positive. 

It  can  be  shown  straightforwardly  that  R^(&)  is  a  convex  U  func- 
tion  of  5  as  C-allager  has  shown  for  the  scalar  source  and  distortion 
measure  case  [1,  pp.  445-446]. 

Since  the  argument  of  the  multiple>constraint  rate  is  a  vector,  the 

inverse  rate-distortion  function  or  distortion-rate  function  will  be  a 

surface  in  m-space  for  each  value  of  its  scalar  argument.  The  distortion 

rate  function  is  given  by  the  surface  ®„(R)  =  iA  :  R„(A)  =  r} .  Since 

l  ^  u  ~  * 

IV  >V 

RtI(A)  is  a  convex  function  of  A,  the  surface  5D„(R)  will  be  the  lower 

~  <v> 

boundary  of  a  convex  region  in  m-space,  i.e.,  if  A^Ag  e  ^.(R),  then 

IV 

+  (1  -  A)Ag  =  ^(R)  for  any  0  5  ^  5  1* 

2 


sEia: 


Although  Ry(5)  is  defined  for  the  same  source  as  is  H(U),  coding 

theorems  can  be  proved  showing  it  to  be  the  equivalent  entropy  rate 

(rather  than  entropy)  of  a  reproduction  (compressed)  sequence  ju^j  of 

the  i.i.d.  vector  source  sequence  ju^  subject  to  a  vector-valued 

fidelity  criterion.  Such  coding  theorems  are  immediate  extensions  of 

those  for  the  usual  rate  distortion  function  with  a  single  argument  3  , 

(.1,  Section  9.3],  [2,  Chap.  3]  and  hence  are  stated  here  without  proof, 

after  the  necessary  notation  is  introduced. 

Let  | k®  an  i*i*d*  sequence  of  vectors  (n-tuples)  u^  = 

(u,  ,»u,  •  *  ♦» u,  )•  Denote  a  block  of  N  vectors  by  u^  =  (u„,..., 

k,  l  k,  <s  k,  n  ~ 

u„);  the  superscript  N  will  be  suppressed  whenever  possible.  Assume 
~N 

the  distortion  measure  between  blocks  of  vectors  is  a  single-vector  dis¬ 
tortion  measure,  i.e., 


i  8. 

D  (u.  u)  =  *f  >  D(u.  ,<!.  > 

~N~  ~  k4~~K’~k 


where  ^(u^,^)  is  the  per-vector  distortion  measure. 

An  encoder  with  parameters  (N,  M)  is  defined  as  a  mapping 


f£  :  A(U)N  -»  IM  =  (1,2,...,M)  . 


A  decoder  corresponding  to  f_  is  defined  by  the  mapping 

E 


fD  =  h-'Uy  ' 


An  encoder-decoder  is  applied  as  follows:  If  =  i e iM>  then 

/%(N)  a  N 

u  =  f_(i)efi(U)  .  The  encoder-decoder  has  average  distortion  5  if 
E  D„(u,u)  S  5  ,  in  which  case  we  have  a  code  with  parameters  (5, N, M). 
A  nonnegative  number  R  is  called  5-attainable  if  for  arbitrary  e  >  0 


and  N  sufficiently  large,  there  exists  a  code  (p  +  5  e,N,M),  e  =  (e, 

(V  »v  »V  'V 

e,  .  ..,e),  where 


M  <  exp|N[R  +  e]  |  . 


Theorbo  1 :  (Source  Coding  Theorem) 

Given  an  i.i.d.  vector  source  ju^J  and  a  single-vector  vector-valued 

distortion  measure,  then  the  rate  R„(5)  +  6  is  5-attainable . 

u  ~  ~ 


Theorem  2:  (Converse  Source  Coding  Theorem) 


Given  the  source  and  distortion  measure  of  Theorem  1,  no  rate  smaller 
than  Rjj(5)  is  5-attainable.  The  converse  theorem  can  also  be  stated  as 
follows:  If  5  <  ®(R),  then  the  rate  R  is  not  5-attainable. 

<v  rst  /v 

Unlike  as  in  Berger  [2,  Chap.  3),  the  above  theorems  are  stated  in 
terms  of  5-attninable  rates  rather  than  5-admissible  codes.  The  above 
terminology  adapts  more  readily  to  the  coding  theorems  and  examples  con¬ 
sidered  here  and  in  [5]  and  [8] . 

The  evaluation  of  R„(5)  is  in  general  complicated.  The  immediate 

U  ~ 

extension  of  the  Kuhn-Tucker  minimization  of  single  constraint  rate  dis¬ 
tortion  functions  Cl,  p.  459],  [2,  p.  37]  yields 

R„(6)  =  H(U)  +  max  <  £  $<u)  log  f  (p)  -  5  pT 

~  p  —0,  f  e  J(p)  (u€  A(U) 

rw.  rv> 


i 

i 


i 


where  T  stands  for  transpose  and  where  f  =  f(P)  =  {f  (p),  u e  A(U)Je  J(p) 

r+s  *11  ~  ~  ' 

fSJ 

if 


(i)  f(£)  —  0 

(li>  2  fu(£>  <  1,  G  6  fi(U)  . 

usA(U) 


- - rfiiiiaiitfiiiiinMrfi'in  1 1 


1 1 ii-inni  h  1 1 1  W-Jt 


Necessary  and  sufficient  conditions  on  f  to  yield  the  maximum  in  (1) 

»v 

are 

(iii)  that  there  exist  a  nonnegative  solution  u>(u)  to 

fSJ 

^  X 

^  w(u)  fu(jg)  =  Q(u)  ;  ueA(U) 

u  e  fl(U) 

and 

(iv)  that  (ii)  hold  with  equality  for  all  u  such  that 
w(u)  >  0. 

When  A(U)  =  ft(U),  we  can  lower  bound  Ry(S)  by  solving  (ii)  with 
equality  for  all  ueA(U)  and  inserting  the  result  into  (1).  This  f 

/v  (V  ~ 

will  yield  Ry(5)  if  the  auxiliary  conditions  (iii)  and  (iv)  are  satis- 
fied.  The  remaining  maximization  over  P  simply  involves  taking  deriva¬ 
tives  and  will  yield  parametric  expressions  for  Ry(8)  and  5.  The 

A 

A(U)  =  fi(U)  condition  is  crucial  since  only  in  this  case  does  (ii)  with 
equality  become  (| A (U) ||  equations  in  ||A(U)|j  unknowns,  where  ||A(U)|| 
denotes  the  size  of  the  alphabet.  The  properties  of  this  bound — called 
the  extended  Shannon  lower  bound  (ESLB)— are  an  immediate  extension  of 
the  Kuhn-Tucker  minimization  and  are  summarised  below  sans  proof . 

Theorem  3:  Define  the  vector  cp(P)  =  )©  (p).  ueA(U)>  as  the  non- 

■  ■  1  n  rs.  rK/  I  Ty  /+j  w  I 

*  iv  1 

negative  solution  (if  it  exists)  to 


<P(P>  E  =  1  (2a) 

r+t  ****** 


and  p  =  1p,»  •  •  •»  P,,}  is  a  vector  having  nonnegative  entries.  Define 

'v  1  "1  •  M ' 


r5.L>(5)  =  H(U)  + 
u  ~  ~ 


max  /  C 
~  =  ~  (ue  A(U) 


Q(u)  log  cp  (p)  -  8  p 

»v  Vl  ^  »v  <v 


Since  <P(p)  e  5(p)  we  have 


tyS>  s  ruM  <£> 


Furthermore,  (3)  will  hold  with  equality  iff  there  exists  a  nonnegative 
solution  w(fi)  to  (iii)  with  f  =  cp. 

'v  a/  rv 

To  be  properly  careful  we  should  worry  about  the  existence  of  a 
nonnegative  solution  to  the  j| A (U >||  equations  in  ||a(U)II  unknowns  de¬ 
scribed  by  (2).  A  straightforward  extension  of  arguments  in  [6]  based 
on  a  fundamental  result  of  Jelinek  [4]  yield  the  following  sufficient 

condition  for  tightness  of  R  (5)  for  a  region  of  small  5. 

U  ^  ^ 


Theorem  4:  Given  the  vector  source  U,  a  distortion  measure  satisfying 
D(u,u)  >  D(u,u)  =  cp,  and  an  available  reproduction  alphabet  Q(U)  =  A(U), 

ej  (N*7  iv  iv  iv  iv  iv  iv  iv 

there  exists  a  surface  $  (U)  containing  strictly  positive  elements  such 

#N/C 


Rtt<5)  =  R_,  (5)  ,  5  £  B  (U) 

U  ~  U  ~  '  ~  ~c  ~ 


The  surface  B  (U)  is  called  the  cutoff,  or  critical  distortion  surface. 

r%tQ  iv 

As  discussed  in  [8],  the  most  useful  types  of  per-vector  distortion 
measures  are  compound  and  weighted-average  distortion  measures.  The 
joint  rate  with  a  weighted-average  distortion  measure  with  weights 
(a1#  •  • *,an)  can  be  obtained  from  the  joint  rate  with  a  compound  distor¬ 
tion  measure  as  follows:  Consider  a  weighted-average  m-valued  distortion 
measure  with  per-letter  distortion  measures  d^u^,^),  i=l,  ...,n.  We 
have  from  the  definitions  that 


6 


(4a) 


Ry(A>  = 


fA  iAf)  Wrt  Ru(^l’'  * '  ’&? 
(A^  , . . .  ,A^J  e  v(£)  ~ 


:  (4b) 

Since  RyCA^ ,. . .  ,^)  is  a  convex  U  function  of  (Aj^...,^),  Gallager's 
Theorem  4.4.1  [l{  p.  8l]  can  be  extended  to  show  that  the  above  infimum 
occurs  at  the  value  of  {A^ , . . .  ,  say  (A*, .  , .  ,A£),  which  is  in  fi(A) 

and  satisfies  the  following  condition:  Define  A^  =  ^  ^A^  m)’ 


dA  ,  V^l,**',~n)  fA* 

K,j  ~  {A*,..., 


5  ~  Rir(A) 

. &  i  ~~  £ 


k=l , . .  . ,n 


with  equality  for  all  k  such  that  a£  j  >  0.  The  above  condition  can 

*£  M- 

be  abbreviated  to  the  statement  that  the  slopes  of  Ry(A1#  . . .,A^)  in 

»v 

the  coordinate  in  each  of  its  n-vector  arguments  equals  the  slope 

of  R„(A)  in  its  jth  coordinate, 
u  ** 

IV 

Despite  the  apparently  circuitous  way  of  finding  a  vwighted-average 
distortion  measure,  (4)  later  proves  quite  useful  in  [8]  't  shouia  be 
pointed  out  that,  even  though  the  equal-slope  condition  appears  horrendous, 
rate-distortion  functions  are  usually  evaluated  as  parametric  expressions 
for  the  rate  and  distortion  in  terms  of  the  slope.  Hence,  in  actuality, 
this  condition  usually  simplifies  such  evaluations. 


Conditional  Rates 


Given  the  two-dimensional  source  XY  described  by  Q(x,y)  and 
A(XY) ,  the  induced  marginal  source  Y  described  by  Q(y)  and  A(y) , 
and  for  each  yeA(Y)  the  marginal  source  Xy  described  by  Q(x|y) 
and  A(x] y) ,  define  the  conditional  rate-distortion  function  of  X 
given  Y  as 


RXj  Y(^ 


Pt(x|x,y)  e  3*X|  y(A> 


I (X;X|  Y) 


where 


?x|y(A>  =  |  pmf ' s  p  (x|x,y)  :  E  D(x,x)  = 

\  Pt(x|x,y)  Q(x,y)  D(x,x)  g  & 

A 

x,x,y 


The  following  theorem  relates  the  conditional  rate-distortion  function 
to  a  weighted  sum  of  marginal  rate  distortion  functions. 


Theorem  5 


Rx|y(£>  =  Inf  y  Rx|y(VQ<y) 

1  y  e  A(Y) 


where  | ^ (A)  is  the  marginal  rate-  listortion  function  of  the  single 
source  X  ,  and 

y 

0(A)  =  |  [Ay,  y  e  A(Y)  }  :  Ay  Q(y)  =  a| 

'  y  €  A(Y)  ’ 


Proof : 


First  choose  a  set 


|  Ay  |  such  that 

])  £y  =  £ 


-  tcs&tx&b  ■  i  i  f ■ '  nf  ft  ' 


and  then  choose 


||A(Y)||  test  channels  Pt(x|x,y)  e  Px|  y(£y> »  yeA(Y), 
where  P^  y(Ay)  =  jpt<£|x,y)  :  E  D(xfS)  g  aJ.  Then 


y Q(y)  y  d<x.*>  pt(*lx»y>  Q(xly)  =  SQ(y)  ^y  =  ~ 

^  a  y 


y  x,x 


hence  pt(x|x,y)  e  y <£>  •  Thus  for  any  such  set  of  test  channels 


£l(X;X|y)  Q(y)  =  I(X;X|Y)  ^Rx|y<^ 


so  that  choosing  each  test  channel  p  (x|x,y),  y e  A(Y) ,  to  yield 

Rx|y(V'  We  h3Ve 

1  <6) 

y  e  ACY) 

for  any  set  |£y,  yeA(Y)}  satisfying  (5). 

Next  choose  a  test  channel  Pt(x|x,y)e  ^*x j Y (~ > *  This  test  channel 
will  result  in  some  set  of  conditional  distortions  defined  by 


A*  =  %  P+(x|x,y)  Q(x|y)  D(x,x) 


such  that 


<L*  ~y 
ye  A(Y) 


A*  Q(y)  g  A 


For  any  such  test  channel 


I (X ; X |-.Y)  =  ^  I(X;X|y)  Q(y) 

y€A(Y) 


6  I  s<,, 


y  e  A(Y) 


3  ,A  l'  Inl(A,  S  1  W  Q<y> 

{Ve^  yeA(Y) 


Choosing  p  (x|x,  y)  to  yield  R^j^A)  gives 

I 

t 

Ry1v(£>  §  Inf  Y  Ry.  <A  )  W 

X|Y  ~  , 1  c/ an  Z-i  iXj  y  ~y 

;~V  y  e  A(Y) 


which,  with  (6)  proves-  the  theorem. 

i 

Similar  to  (4),  Gallager.’s  Theorpm  4.4.1,  [1,  p*  87]  implies  that 

the  infimum  in  Theorem  5  is  achieved  by  adding  up  the  rate-distortion 

1  5 

functions  at  points  of  equal  slope  in  all  coordinates,  i.e., 


R„u,(4> 


-I 


y  e  A(Y) 


RX|y'4>Q<y) 


where 


(Ay)  «  M. 


and  for  each  k=l, ...,m 


S|y<Sy>  °  1x|y~’  ’’  y£A<v)' 


8;  ’  , 


In  general  X  and  Y  may  themselves  be  vector  sources.  For  simplicity, 

i  ; 

however,  we  shall  here  rjemain  with  scalar  notation. 


The  appropriate  coding  theorems  for  conditional  rate-distortion 
functions  are  slightly  more  complicated  to  state  than  Theorem  1  due  to 
the  presence  of  the  side  information.  Seme  of  the  definitions  must  be 
modified  accordingly.  An  encoder  with  parameters  (n,M)  is  now  defined 
as  a  mapping 

fE  :  A «»)"  -  I„ 

A  decoder  corresponding  to  f„  is  defined  by  the  mapping 
■|,"T1  ■  E 

f D  :IMX  A(Y)H  ->  n(X)n  . 

An  encoder-decoder  with  parameters  <n,M)  is  applied  as  follows:  Let 

f  (x,y)  =  i  e  I  ,  then  2  =  1  (i,y).  The  encoder-decoder  has  average 
E  ~  M  ^  D  ^ 

distortion  A  if 

rst 

E  |D(x,$)|  S  A 

M  V  M  M  f  /W 

where  the  expectation  is  now  over  the  joint  ensemble  XY.  In  such  a 
case  we  have  a  code  (A,  n,  M).  As  before,  a  rate  R  is  said  to  be 

i 

A-attainable  (conditioned  on  |  |  >  if  for  arbitrary  e  >0  and  n- 

sufficiently  large,  there  exists  a  code  (A  +  e,  n,  M)  where 

M  ^  expjnfR  +  e]| 

Theorem  6:  (Conditional  Source  Coding  Theorem) 

Given  an  i.i.d.  sequence  of  dependent  pairs  |(xk,y^)J  and  a  single¬ 
vector  distortion  measure,  assume  that  both  encoder  and  decoder  are  al¬ 
lowed  to  observe  perfectly  the  sequence  |yk|,  Then,  the  rate  R^j^fA) 
is  A-attainable,  and  no  rate  smaller  than  R^iy(A)  is  A-attainable. 


T 


The  above  theorem  can  be  proved  using  a  fairly  straightforward 
extension  of  the  usual  techniques  [2,  Section  3.2],  We  get  a  coding 
theorem  for  free,  however,  by  noting  that  (7)  is  almc3t  identical  to 
Berger’s  (6.1.21)  [2,  p.  184]  and  hence  is  the  (multiple-constraint) 
rate-distortion  function  of  an  i.i.d.  composite  source  with  "switch" 
pmf  Q(y )  and  ||A(Y)||  subsource  pmf's  Q(xjy),  yeA(Y). 

Note  the  obvious  similarity  between  (7)  and  the  corresponding 
entropy  relation 

H(X|Y)  =  H(xj  y)Q(y) 

y  e  A(Y) 

One  implicit  difficulty  with  conditional  rates  is  the  choice  of  the 
appropriate  reproducing  alphabet  for  each  yeA(Y).  There  are  two 
natural  choices — either  the  corresponding  conditional  source  alphabet 
A(xly)  or  the  full  alphabet 

A(X)  =  U  A(X| y) . 
y  e  A(Y) 

For  greatest  ease  in  evaluating  R^|y(A)  it  is  desirable  to  have  iden¬ 
tical  source  and  available  reproducing  alphabets,  as  previously  noted. 
Specifically,  the  ESLB  is  well  defined  only  for  this  case.  Thus,  the 
happiest  possible  state  of  affairs  would  be  if  ft(X|y)  =  A(x|y)  for 
each  y.  Unfortunately,  however,  the  assumption  usually  required  is  that 

A  . 

the  reproducing  alphabet  n(X|y)  be  A(X)  for  each  y.  Thus,  in  gen¬ 
eral,  we  may  have 

A(X| y)  %  fl(Xjy)  =  A(X) 
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and  hence  actual  evaluations  of  R^j^A)  may  be  quite  complicated  be¬ 
cause  the  matrix  E  of  (2b)  is  not  square  and  not  invertible. 

If  A(X)  =  A(x|y),  all  yeA(Y),  then  R^^CA)  usually  can  be 
evaluated  straightforwardly.  An  equivalent  assumption  is  that  the  alpha¬ 
bet  A(XY)  is  the  cartesian  product  of  the  marginal  alphabets,  i.e., 
A(XY)  =  A(X)  x  A(Y).  In  order  to  obtain  some  reasonably  general  bounds 
and  evaluations  for  the  rate-distortion  functions  in  the  theorems,  we 
occasionally  assume  that  A(XY)  =  A(X)  x  A(Y).  This  assumption  is  not 
made  in  the  more  general  theorems.  IF  A(XY)  =  A(X)  XA(Y)  the  ESLB  of 


v 


r(A)  is  well  defined  as  the  weighted  sum  of  the  individual  ESLB's: 


Ry  |  v  (A)  =  Inf 

'  |AU$(A)  „ 

1  ~yi  ~  y 


I  h[j% 


)  Q(y) 


-  Int  ^  Q(y)  = 


(L) 

where  R^j^A)  is  given  by  (2)  with  U  =  X^.  The  infimum  is  obtained 
by  adding  up  the  functions  at  points  of  equal  slope  in  each  component. 
As  in  Theorem  4,  we  have  the  following  result: 


Corollary  1:  If  A(XY)  =  A(X)  X  A(Y)  and  (4)  is  satisfied,  then  there 
exists  a  cutoff  distortion  surface  2)  (x|y)  >0  such  that 


|Y <~>  =  £  =  £c(xLy)  • 


Difference  Distortion  Measures  and  Examples 

The  calculations  involved  in  evaluating  conditional  rates  are  often 
simpler  when  dealing  with  difference  distortion  measures.  Roughly 
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speaking,  given  a  conditional  rate-distortion  function  and  a  difference 
distortion  measure,  there  exists  a  simple  upper  bound  which  actually 
yields  the  conditional  rate  iff  the  source  satisfies  a  certain  property. 


Theorem  7:  Let  a  product  ensemble  XY  and  a  difference  distortion 
measure  be  given.  Then  for  any  function  f(y)  defined  on  A(Y) 


RX]Y(^  =RX-f(Y)|Y^  =^X-f<Y)v* 


(A)  gR, 


with  equality  iff  y  and  x  -  f(y)  are  independent. 


Proof :  Define  z  =  x  -  f(y).  Given  a  source  pmf  ®x|Y^xly^  we  wilJ 
have  also  the  source  pmf 

QZ|Y<zly)  =  ^x|y(z  +  f  (y>|y> 

since  x  =  z  +  f(y).  Furthermore,  any  test  channel  p^^xlx.y)  e  ?X|y(A) 

induces  a  test  channel  p*2\z|z,y)  =  p^^(z  +  f(y)|z  +  f(y),y)  e  V.  (A) 

t  t  Z|y  ~ 


since 


E|d(z,y)|y|  =  ^  d(z  -  z)  p*2)(zlz,y)  Qz!y(z!y) 

z  e A(zjy)  1 

z  e  fi(2|y ) 


^  d(x  -  x)  p! 1  ^ (x |x, y )  Q  |  (x|y)  ^  A 
eA(x|y)  1  X|Y 


2  e  n(x|y> 

For  the  test  channel  p^(x|x,y)  we  have 

I  (X;  X  |y )  =  H(X|y)  -  K(X|Xy)  =  H(X  +  f(y)jy)  -  H(X  +  f(y)|X  +  f(y),y) 


=  I(Z;Zjy) 
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Choosing  p^*\$|x,y)  to  yield  R^, ,  (i)  we  have 

«x|,(a)  =  tczjKly)  iRz|y(a>  (8) 

By  defining  x  =  z  +  f(y)  and  repeating  the  previous  procedure  with 
and  z  interchanged  we  obtain 

8x|y<S)SlV£)  <9> 

Eq.  (9)  implies  the  left  hand  equality  in  Theorem  7.  The  right  hand 
inequality  follows  from  theorem  2.1  of  [8]. 

This  theorem  can  be  used  to  provide  an  alternate  evaluation  of  the 
conditional  rates  in  the  examples  of  [8].  In  the  binary  example  f(y)  =  y 
and  the  source  x  can  be  viewed  as  the  mod-2  sum  of  two  independent 
random  variables  y  and  z  where 

i 

Q(y)  =  y  =  o,  l 

«(z)  =  <1  -  p)^'°  p1'5*'0  i 

The  Gaussian  case  can  be  viewed  similarly.  Perhaps  a  more  interesting 
view  of  the  Gaussian  case  is  to  choose  the  function  f(y)  such  that 
x  -  f(y)  and  y  are  independent.  It  is  well  known  that  this  is  ac¬ 
complished  by  choosing  the  conditional  expectation 


f  (y)  =  E  x|y  =  +  r  ~  (y  -  n^) 


RjjjyCA)  =  Rx_f(Y)^^  is  simP1y  the  marginal 


rate-distortion  func- 


2  2 

tion  of  a  zero  mean  Gaussian  random  variable  with  variance  a  (1  -  r  )• 
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It  is  worth  observing  that  in  both  of  the  preceding  examples  the 
conditional  rate  of  x  given  y  is  the  marginal  rate  of  the  "innova¬ 
tion"  v  =  x  -  x(y),  where  x(y)  is  the  best  estimate  of  x  given  y, 
i.e.f  the  estimate  that  minimizes  E  d (x, x(y ) ) . 


L.W.'T-fa 
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